In a network of communicating machines having relatively large numbers of devices, such as for example an enterprise network or other computer network, the amount of computing assets, and the amount of available data for use by those computing assets, grows rapidly with the size of that network. Managing such networks, including obtaining information regarding those assets and data, and modifying allocation of those assets and data, can be a Herculean task, often requiring substantial computation and communication.
One known problem in the art is that the computation and communication needed to manage such a network is often not readily available, and even if available, would take relatively large amounts of time to perform management operations. This has the effect that latency in providing answers to questions about the network, or in performing modifications to the network, is relatively high, with the effect that real-time management of such networks can be infeasible when the network is relatively large. It typically takes more time to collect information about the status of devices coupled to the network than it takes for those statuses to change, making collection of that information impractical, inaccurate, untimely, and otherwise unsatisfactory. It typically takes a relatively large amount of communication and processing resources to bring that information to the attention of a network operator, such as someone operating a network control server.
Known systems attempt to ameliorate these issues by performing some degree of aggregation or processing of data at intermediate control levels, with the effect that there might be several levels of indirection before information from a device is finally delivered to the network control server. These techniques do not scale well, so that a network with (say) 100,000 nodes would likely still take several hours or more to report the status of individual ones of those nodes, or even of an aggregate thereof. In that time, many nodes will likely have changed their status, making that report obsolete.
Known systems also amass information about network devices into one or more relatively large databases, so that network operators can query those databases for information about the network. These techniques also do not scale well, so that a relatively large network would have enough interesting data to swamp the operations of that database. One result is that only relatively few database queries can be made within resource limits of the database or its servers.